chore: add basic benchmarking by mattsb42-aws · Pull Request #275 · aws/aws-encryption-sdk-python

mattsb42-aws · 2020-06-11T21:34:34Z

Description of changes:

This adds some tests that benchmark the performance of the client under different configurations.

Results from running this on an m5a.xlarge EC2 instance:

---------------------------------------------------------------------------------------------------------------------------------- benchmark: 54 tests -----------------------------------------------------------------------------------------------------------------------------------
Name (time in ms)                                                                                                          Min                 Max                Mean            StdDev              Median               IQR            Outliers       OPS            Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_end2end_performance[decrypt only-single frame-cache and only miss once-AWS KMS master key provider]                2.8315 (1.0)        3.7074 (1.22)       2.9646 (1.0)      0.2633 (9.59)       2.8719 (1.0)      0.0644 (2.98)          1;1  337.3181 (1.0)          10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw RSA master key]                          2.8418 (1.00)       3.3037 (1.09)       3.0307 (1.02)     0.1612 (5.87)       3.0182 (1.05)     0.2963 (13.69)         4;0  329.9582 (0.98)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw AES master key]                          2.8661 (1.01)       4.1578 (1.37)       3.0152 (1.02)     0.4016 (14.62)      2.8940 (1.01)     0.0216 (1.0)           1;1  331.6482 (0.98)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-AWS KMS master key provider]                 2.9036 (1.03)       3.7566 (1.24)       3.0272 (1.02)     0.2578 (9.39)       2.9545 (1.03)     0.0577 (2.66)          1;1  330.3333 (0.98)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw RSA keyring]                             2.9234 (1.03)       3.0458 (1.00)       2.9720 (1.00)     0.0367 (1.34)       2.9756 (1.04)     0.0417 (1.93)          3;0  336.4700 (1.00)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw RSA master key]                         2.9325 (1.04)       3.1961 (1.05)       3.0515 (1.03)     0.0879 (3.20)       3.0451 (1.06)     0.1453 (6.71)          3;0  327.7041 (0.97)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw AES master key]                         2.9329 (1.04)       4.2640 (1.40)       3.1076 (1.05)     0.4070 (14.82)      2.9892 (1.04)     0.0373 (1.72)          1;1  321.7916 (0.95)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-AWS KMS keyring]                            2.9513 (1.04)       4.3001 (1.41)       3.2733 (1.10)     0.4726 (17.21)      3.0528 (1.06)     0.2174 (10.05)         2;2  305.5039 (0.91)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-AWS KMS master key provider]                2.9515 (1.04)       6.3232 (2.08)       3.4855 (1.18)     1.0961 (39.91)      3.0233 (1.05)     0.0720 (3.33)          1;2  286.9019 (0.85)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-Raw AES keyring]                             2.9519 (1.04)       3.0399 (1.0)        2.9925 (1.01)     0.0308 (1.12)       2.9969 (1.04)     0.0605 (2.79)          5;0  334.1644 (0.99)         10         100
test_end2end_performance[decrypt only-single frame-cache and miss every 10-AWS KMS keyring]                             2.9543 (1.04)       5.3401 (1.76)       3.2993 (1.11)     0.7188 (26.17)      3.0880 (1.08)     0.0655 (3.03)          1;2  303.0971 (0.90)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw RSA keyring]                            2.9677 (1.05)       3.1697 (1.04)       3.0521 (1.03)     0.0616 (2.24)       3.0329 (1.06)     0.0711 (3.29)          3;0  327.6395 (0.97)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw AES master key]                         2.9687 (1.05)       4.3998 (1.45)       3.1586 (1.07)     0.4379 (15.94)      3.0413 (1.06)     0.0896 (4.14)          1;1  316.5959 (0.94)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw RSA master key]                         2.9696 (1.05)       4.3351 (1.43)       3.1453 (1.06)     0.4198 (15.28)      3.0185 (1.05)     0.0698 (3.23)          1;1  317.9318 (0.94)         10         100
test_end2end_performance[decrypt only-single frame-cache and only miss once-Raw AES keyring]                            3.0199 (1.07)       3.1493 (1.04)       3.0767 (1.04)     0.0409 (1.49)       3.0693 (1.07)     0.0512 (2.37)          4;0  325.0214 (0.96)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-AWS KMS keyring]                            3.0994 (1.09)       6.3785 (2.10)       3.4788 (1.17)     1.0192 (37.11)      3.1612 (1.10)     0.0224 (1.03)          1;2  287.4553 (0.85)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw AES keyring]                            3.1026 (1.10)       3.3538 (1.10)       3.2067 (1.08)     0.0719 (2.62)       3.1940 (1.11)     0.0457 (2.11)          3;3  311.8437 (0.92)         10         100
test_end2end_performance[encrypt only-single frame-cache and only miss once-Raw RSA keyring]                            3.1123 (1.10)       3.2192 (1.06)       3.1422 (1.06)     0.0350 (1.27)       3.1319 (1.09)     0.0505 (2.33)          1;0  318.2455 (0.94)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw AES master key]                          3.1880 (1.13)       3.2767 (1.08)       3.2296 (1.09)     0.0275 (1.0)        3.2315 (1.13)     0.0432 (1.99)          3;0  309.6329 (0.92)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw RSA keyring]                             3.3358 (1.18)       4.6336 (1.52)       3.5028 (1.18)     0.3986 (14.51)      3.3682 (1.17)     0.0427 (1.97)          1;1  285.4823 (0.85)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw AES keyring]                             3.3393 (1.18)       3.4384 (1.13)       3.3837 (1.14)     0.0350 (1.27)       3.3823 (1.18)     0.0668 (3.09)          4;0  295.5379 (0.88)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-Raw RSA master key]                          3.3598 (1.19)       3.4824 (1.15)       3.4288 (1.16)     0.0447 (1.63)       3.4377 (1.20)     0.0866 (4.00)          5;0  291.6510 (0.86)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw AES keyring]                                            4.3160 (1.52)       5.7662 (1.90)       4.6448 (1.57)     0.4091 (14.89)      4.5205 (1.57)     0.1211 (5.60)          1;1  215.2967 (0.64)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw AES master key]                                         4.3175 (1.52)       4.4212 (1.45)       4.3650 (1.47)     0.0318 (1.16)       4.3632 (1.52)     0.0408 (1.88)          3;0  229.0965 (0.68)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw AES master key]                                         4.6765 (1.65)       4.7871 (1.57)       4.7278 (1.59)     0.0323 (1.18)       4.7291 (1.65)     0.0331 (1.53)          3;1  211.5170 (0.63)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw AES keyring]                                            4.7414 (1.67)       6.1184 (2.01)       4.9875 (1.68)     0.4014 (14.61)      4.8880 (1.70)     0.0801 (3.70)          1;1  200.5015 (0.59)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw RSA keyring]                                            4.7542 (1.68)       6.0527 (1.99)       4.9270 (1.66)     0.3972 (14.46)      4.7965 (1.67)     0.0658 (3.04)          1;1  202.9649 (0.60)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-AWS KMS master key provider]       6.1176 (2.16)      10.2705 (3.38)       6.8095 (2.30)     1.2904 (46.98)      6.3088 (2.20)     0.5383 (24.87)         1;2  146.8546 (0.44)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw RSA master key]                6.3366 (2.24)       8.1186 (2.67)       6.7200 (2.27)     0.5168 (18.81)      6.6344 (2.31)     0.3505 (16.19)         1;1  148.8095 (0.44)         10         100
test_end2end_performance[encrypt only-single frame-no cache-Raw RSA master key]                                         6.3942 (2.26)       7.8302 (2.58)       6.6359 (2.24)     0.4263 (15.52)      6.5032 (2.26)     0.1329 (6.14)          1;1  150.6965 (0.45)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-AWS KMS keyring]                   6.4201 (2.27)      10.5999 (3.49)       7.0723 (2.39)     1.3027 (47.43)      6.5317 (2.27)     0.4896 (22.62)         1;2  141.3960 (0.42)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw AES master key]                 6.4928 (2.29)       8.7652 (2.88)       6.9987 (2.36)     0.6680 (24.32)      6.8791 (2.40)     0.5484 (25.34)         1;1  142.8832 (0.42)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw AES master key]                6.6806 (2.36)       8.4195 (2.77)       6.9204 (2.33)     0.5300 (19.30)      6.7670 (2.36)     0.1381 (6.38)          1;1  144.5011 (0.43)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw AES keyring]                    6.7439 (2.38)       8.5047 (2.80)       7.3200 (2.47)     0.4683 (17.05)      7.2874 (2.54)     0.3691 (17.05)         2;1  136.6125 (0.40)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw RSA keyring]                   6.8581 (2.42)       7.1733 (2.36)       6.9659 (2.35)     0.0851 (3.10)       6.9623 (2.42)     0.0587 (2.71)          2;1  143.5571 (0.43)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and only miss once-Raw AES keyring]                   6.9545 (2.46)       7.3742 (2.43)       7.1091 (2.40)     0.1218 (4.43)       7.0802 (2.47)     0.1478 (6.83)          3;0  140.6640 (0.42)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw RSA master key]                 7.1794 (2.54)       8.5197 (2.80)       7.3895 (2.49)     0.4025 (14.65)      7.2523 (2.53)     0.1358 (6.27)          1;1  135.3266 (0.40)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-Raw RSA keyring]                    7.1805 (2.54)       8.2461 (2.71)       7.7462 (2.61)     0.4923 (17.93)      7.7774 (2.71)     0.9414 (43.49)         2;0  129.0961 (0.38)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw AES master key]                                8.9643 (3.17)      10.7343 (3.53)       9.3247 (3.15)     0.5125 (18.66)      9.2113 (3.21)     0.2976 (13.75)         1;1  107.2416 (0.32)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw AES keyring]                                   9.1484 (3.23)      10.6542 (3.50)       9.4189 (3.18)     0.4414 (16.07)      9.2794 (3.23)     0.1723 (7.96)          1;1  106.1698 (0.31)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw RSA master key]                                        11.2803 (3.98)      12.7799 (4.20)      11.5652 (3.90)     0.4365 (15.89)     11.4336 (3.98)     0.1646 (7.61)          1;1   86.4666 (0.26)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-AWS KMS master key provider]                11.3130 (4.00)      13.6719 (4.50)      11.7862 (3.98)     0.7034 (25.61)     11.4614 (3.99)     0.4695 (21.69)         1;1   84.8450 (0.25)         10         100
test_end2end_performance[decrypt only-single frame-no cache-Raw RSA keyring]                                           11.5758 (4.09)      11.7989 (3.88)      11.6848 (3.94)     0.0755 (2.75)      11.6830 (4.07)     0.0991 (4.58)          4;0   85.5812 (0.25)         10         100
test_end2end_performance[encrypt only-single frame-cache and miss every 10-AWS KMS keyring]                            11.6115 (4.10)      14.5919 (4.80)      12.4075 (4.19)     0.8815 (32.10)     12.2145 (4.25)     0.6012 (27.78)         1;1   80.5965 (0.24)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw RSA keyring]                                  16.2953 (5.75)      18.4330 (6.06)      16.8372 (5.68)     0.5949 (21.66)     16.6533 (5.80)     0.2282 (10.54)         1;1   59.3921 (0.18)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-Raw RSA master key]                               17.8754 (6.31)      19.5608 (6.43)      18.2335 (6.15)     0.5187 (18.89)     18.0378 (6.28)     0.2469 (11.41)         1;2   54.8440 (0.16)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-AWS KMS keyring]                   22.6073 (7.98)      27.0973 (8.91)      24.2102 (8.17)     1.3239 (48.20)     24.1547 (8.41)     1.6790 (77.57)         2;0   41.3049 (0.12)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-cache and miss every 10-AWS KMS master key provider]       22.9630 (8.11)      26.5351 (8.73)      24.2309 (8.17)     1.1369 (41.39)     23.9474 (8.34)     1.6900 (78.08)         3;0   41.2696 (0.12)         10         100
test_end2end_performance[decrypt only-single frame-no cache-AWS KMS master key provider]                               85.9302 (30.35)     97.9246 (32.21)     92.6231 (31.24)    4.2571 (155.00)    92.3338 (32.15)    6.3445 (293.13)        5;0   10.7964 (0.03)         10         100
test_end2end_performance[decrypt only-single frame-no cache-AWS KMS keyring]                                           86.0042 (30.37)     98.2628 (32.32)     92.0426 (31.05)    4.0520 (147.53)    91.9101 (32.00)    3.2239 (148.95)        4;0   10.8645 (0.03)         10         100
test_end2end_performance[encrypt only-single frame-no cache-AWS KMS keyring]                                           87.1333 (30.77)    100.9260 (33.20)     94.6055 (31.91)    4.0725 (148.28)    93.4407 (32.54)    5.4352 (251.12)        3;0   10.5702 (0.03)         10         100
test_end2end_performance[encrypt only-single frame-no cache-AWS KMS master key provider]                               88.1433 (31.13)    101.6990 (33.46)     93.3860 (31.50)    4.0176 (146.28)    93.2626 (32.47)    4.9964 (230.85)        3;0   10.7082 (0.03)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-AWS KMS master key provider]                     180.7815 (63.85)    194.7949 (64.08)    186.9117 (63.05)    4.7156 (171.69)   186.0659 (64.79)    6.3366 (292.76)        3;0    5.3501 (0.02)         10         100
test_end2end_performance[encrypt decrypt cycle-single frame-no cache-AWS KMS keyring]                                 184.4317 (65.13)    195.0076 (64.15)    188.8441 (63.70)    3.6701 (133.63)   188.4976 (65.64)    5.0244 (232.14)        4;0    5.2954 (0.02)         10         100
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Check any applicable:

Were any files moved? Moving files changes their URL, which breaks all hyperlinks to the files.

…ically only mark KMS tests as integ

…ues and remove 1kiB-frame benchmark test

mattsb42-aws · 2020-06-12T19:15:35Z

Looks like breaking CI is due to pypa/virtualenv#1857

texastony · 2022-09-16T16:57:43Z

Putting in Draft as this is for keying which has not shipped and is not currently on our roadmap

mattsb42-aws · 2022-09-21T01:38:10Z

You do you, but IMO having benchmarking data in the tests would be useful without keyrings. I'd be happy to take a look at stripping the keyring benchmarks out of this.

texastony · 2022-09-21T18:20:33Z

Oh... @mattsb42-aws it appears the mattsb42-aws:benchmarking has been deleted... maybe I can download this source code via gh?

texastony · 2022-09-21T18:44:21Z

I have recreated this PR here.
I need to replace the Keyring references with Master Key Provider references.
@mattsb42-aws If you want, you can pitch in for that work.
Otherwise, I will get to it eventually...

mattsb42-aws · 2022-10-04T17:13:42Z

Whoops; I guess I got a bit overzealous deleting forks when I changed teams. I'll see about dusting off my memory of how I put this together to re-arrange it to use MKPs.

mattsb42-aws added 10 commits June 11, 2020 12:49

chore: fix typo

a259cf7

chore: add end-to-end benchmark tests

db1b270

chore: prettify yaml

e2037f2

chore: add CPython 3.8 benchmark run to Travis

f0c0234

chore: simplify test configuration

1de1733

chore: reorganize benchmark files into their own directory and specif…

bcf4723

…ically only mark KMS tests as integ

chore: change parameterized tests to indirect references to large val…

30ad9c7

…ues and remove 1kiB-frame benchmark test

chore: split local and integ benchmarks

877057d

chore: fix algorithms benchmark to actually change algorithms

b404f2a

chore: remove unnecessary strategy section for benchmarks job

a4352b0

chore: rename tox benchmark environments to avoid conflicts

873218c

texastony marked this pull request as draft September 16, 2022 16:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add basic benchmarking#275

chore: add basic benchmarking#275
mattsb42-aws wants to merge 11 commits intoaws:keyringfrom
mattsb42-aws:benchmarking

mattsb42-aws commented Jun 11, 2020

Uh oh!

mattsb42-aws commented Jun 12, 2020

Uh oh!

texastony commented Sep 16, 2022

Uh oh!

mattsb42-aws commented Sep 21, 2022

Uh oh!

texastony commented Sep 21, 2022

Uh oh!

texastony commented Sep 21, 2022

Uh oh!

mattsb42-aws commented Oct 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mattsb42-aws commented Jun 11, 2020

Check any applicable:

Uh oh!

mattsb42-aws commented Jun 12, 2020

Uh oh!

texastony commented Sep 16, 2022

Uh oh!

mattsb42-aws commented Sep 21, 2022

Uh oh!

texastony commented Sep 21, 2022

Uh oh!

texastony commented Sep 21, 2022

Uh oh!

mattsb42-aws commented Oct 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants